Beyond Class A: A Proposal for Automatic Evaluation of Discourse

Introduct ion The DARPA Spoken Language communi ty has just completed the first trial evaluation of spontaneous query/response pairs in the Air Travel (ATIS) domain. 1 Our goal has been to find a methodology for evaluating correct responses to user queries. To this end, we agreed, for the first trial evaluation, to constrain the problem in several ways: D a t a b a s e A p p l i c a t i o n : Constrain the application to a database query application, to ease the burden of a) constructing the back-end, and b) determining correct responses; C a n o n i c a l A n s w e r : Constrain answer comparison to a minimal "canonical answer" that imposes the fewest constraints on the form of system response displayed to a user at each site; T y p e d I n p u t : Constrain the evaluation to typed input only; Class A: Constrain the test set to single unambiguous intelligible utterances taken without context that have well-defined database answers ("class A" sentences). These were reasonable constraints to impose on the first trial evaluation. However, it is clear that we need to loosen these constraints to obtain a more realistic evaluation of spoken language systems. The purpose of this paper is to suggest how we can move beyond evaluation of class A sentences to an evaluation of connected dialogue, including out-of-domain queries.